Skip to content

FE-1583: Run the connected optimizer's trials through the experiments backend - #9516

Merged
kube merged 18 commits into
claude/web-optimizer-python-corefrom
claude/web-optimizer-react-channel
Sep 8, 2026
Merged

FE-1583: Run the connected optimizer's trials through the experiments backend#9516
kube merged 18 commits into
claude/web-optimizer-python-corefrom
claude/web-optimizer-react-channel

Conversation

@kube

@kube kube commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Important

Experimental
Behind the In-browser optimization feature flag.

Summary

Before this PR, PetrinautOptimizationContext accepted a self-contained remote capability, and a study's trials were numbers in a table. Study drawer's Surface could sample the objective locally, but it knew nothing of where the optimizer was looking, and the optimizer could not use the browser's compute at all.

This PR lets the context carry a connected optimization source and runs its trials through the experiments layer. A new runDetachedObjective action runs a frozen study's objective at one parameter point on the chosen backend with pinned seeds and streams its frames. Optimizations provider evaluates each trial through it, follows the trial in the navigator, streams its metrics live, and after the study settles refines the best point, or whichever point the user picks. Study drawer shows the optimizer's own steps as dots on the Surface with the field filling in as steps report, a Stop that keeps the study and a Continue that adds steps to it, up to four steps in flight with an activity list, and one metric line always in view. Form gains Runs per step, Parallel steps and the experiments' Backend switch. A remote capability such as the HASH host's NodeAPI bridge is unaffected, and the connected source only takes effect with the experimental setting on.

9516.mp4

Links

Changes

Feature flag

  • New experimental setting In-browser optimization under Settings > Simulation

    Row shows only when the host supplies a connected optimizer.

  • useOptimizationSource is the one place that reads the context and the setting

    A remote capability passes through as is.
    A connected source passes through only with the setting on, else the tab stays hidden and nothing connects; turning it off cancels connected runs and disposes.

Experiments layer

  • runDetachedObjective runs a study's objective at one point and streams it

    Compiles once per study, chooses the backend once per study and backend with the experiments' registrations at a third of the CPU pool, pins per-run seeds on the CPU pool only, streams frames and progress.
    Settles with per-run results or a reason: compile diagnostics, backend refusals, failed runs, cancellation.

  • Runs queue per queueKey while sharing one compiled snapshot per cacheKey

    Parallel steps of one study overlap on the backend; refinement rungs of one point stay in order.

  • A cancelled run settles instead of hanging its queue

    Instantiation gets its own abort controller, mirrored from the run signal only until the backend handle exists.
    A later cancel goes through the handle, whose cancelled event releases the queue to the next run.

  • Sweep throttle moves to experiments/shared for reuse

Optimization channel and provider

  • Channel evaluates a trial as one detached objective run with the trial's seeds and the study's backend

    Objective is the mean of the per-run finals when the backend reports them, else the last frames' value; a failure prunes the trial with its reason; the channel never throws.

  • Records carry computeBackend, the study's axes and, for a connected study only, a connected state with its navigation, selection stream, activity, inFlight trials, resumable, parallelism and fallback reason

    Remote records carry connected: null, the one discriminant the drawer branches on.
    Navigation follows the most recently started step; a user move stops following. Selection stream carries the followed step's frames, then the refined point's.

  • Study settles on its best point when it completes or stops

    Navigation moves to the best step while it still follows, and refinement starts there.

  • Point refinement climbs the run ladder to 100 runs for the navigated point

    Starts when the study settles or the user moves, cancels and restarts on a navigation change, caches per point, stops with the reason on failure.
    Between rungs it compares the point's estimate with the best: a point that cannot beat the best by 2.5 standard errors stops after its first rung with the note "8 runs · cannot beat the best"; the best point refines fully.

  • extendOptimization(id, steps) continues a finished or stopped connected study

    Provider calls extendOptimizationRun, re-attaches from the record's cursor, turns following back on; the steps continue Optuna's numbering.
    Stop keeps the attachment so the segment's terminal event lands and marks the record resumable.

  • Removing a connected record releases its study in the worker
  • Connected runs are not written to session storage since they cannot outlive the page

Form and drawer

  • Create form gains Runs per step, Parallel steps, Seed and the Backend switch

    Runs per step lands in execution.seedsPerTrial; the total-steps rule multiplies by it.
    Parallel steps is 1 to 4 and shows for a connected source only.
    Seed starts at a fresh random value each time the form opens and seeds both the proposals and the runs; every study used to share one fixed seed, so two studies drew the same random steps.
    Backend cell is the experiments' shared toggle, shown for a connected source with WebGPU on; it stays disabled with the reason because the GPU backend cannot compute an expression objective.

  • Connected study drawer lays everything out to fit one screen

    A summary strip carries status, finished over requested steps, best value, the backend badge, the steps bar and the "N computing" chip.
    Parameters band holds a slider per axis, Follow steps and the refinement status.
    Surface sits beside the objective chart, the Steps table fills the rest, and the footer holds Stop, or Remove and Continue with a step count.

  • Surface shows the optimizer's steps only

    Each step with an objective is one sample of the field at its projected position and a dot, the best emphasized, pruned steps hollow.
    A step in flight enters the field with its running value, and so does the refined point after settle, so the plot fills in as the study streams.
    While a running study is followed the plot and the sliders are display-only; once the study is terminal, or Follow steps is off, a click or drag commits the navigation.

  • Remote study drawer keeps its locally sampled Surface and its Cancel
  • New layers react.optimizations.connected-study and ui.views.editor.optimizations declare the study machinery and the Optimizations tab in the architecture docs
  • Shared pieces move to SimulateView/shared: useGpuAvailability, ComputeBackendToggle, ComputeBackendBadge, MetricTiles, ComputeActivity, SummaryStrip, format-value

    One number formatter for the list, the drawer and the experiments; one batch registry in experiments/shared for sweeps and studies; one errorMessage helper; the connected and remote drawers render the same summary strip.

  • Slider in @hashintel/ds-components accepts disabled

Stories and example

  • Storybook Simulate / Browser optimizer (real) runs the real Pyodide optimizer against the real experiments backends

    SirCpu, SirGpuRequested, SupplyChain, VaccinationCampaign start a study on load; Manual opens the create form.
    Shared harness simulate-view-story-harness.tsx with AutoStudy; the synthetic-optimizer story is renamed RunSupplyChainOptimizationSyntheticOptimizer.

  • New core example Vaccination Campaign built for optimization demos

    SIR with pre-wave vaccination coverage and contact reduction, a Total cost objective whose minimum lies inside the domain, GPU-eligible; listed under Load example.

Review fixes

  • Study seed masks a 32-bit crypto draw with the seed range instead of reducing it modulo
  • Stop on a connected study whose run has no id yet aborts the creation

    The run is cancelled where it was made once its id arrives, and nothing attaches to it.

  • A point stopped early as unable to beat the best resumes its ladder once it is the best

    A parked point is re-evaluated when a trial changes the best, and when the study settles.

  • Surface cell cache keys on the axes as well as the positions

    Swapped X and Y axes never reuse each other's samples.

  • One changeset per package

    @hashintel/petrinaut, @hashintel/ds-components and one for the Vaccination Campaign example in @hashintel/petrinaut-core.

  • Continue follows the terminal event's resumable flag

    A study stopped while the runtime loads, or while another study holds the worker, has no study to continue and offers no Continue.

Known issues

  • GPU studies are unreachable from the form until the WebGPU backend computes expression metrics

    Record, channel and registrations already carry the choice.

Test coverage

  • provider/detached-objective.test.ts:

    Streaming, pinned seeds on the CPU and none on the GPU, fallback re-pinning, failure reasons for refusal, errored runs and compile errors, one walk per study and backend, per-key queueing, a cancelled batch settling and releasing the queue, distinct queue keys side by side, cancel and dispose.

  • optimizations/provider.test.tsx, provider/connected-study.test.ts, provider/point-refinement.test.ts, provider/point-refinement/objective-estimate.test.ts, provider/activity-registry.test.ts, channel/create-optimization-channel.test.ts:

    Setting gate on, off and toggled, remote unaffected, trials through runDetachedObjective with seeds and backend, following and user moves, settle on the best, the ladder and its early stop, restart on navigation change, stop then continue from the cursor, refused continuation, removal releasing the study, activity throttling.

  • create-optimization-drawer.test.tsx, view-optimization-drawer.test.tsx, study-summary-strip.test.ts, steps-table.test.ts, optimization-navigator.test.tsx, optimization-surface/surface-plot.test.tsx, simulate-view.test.tsx, viewport-settings-dialog.test.tsx:

    Runs per step, parallel steps, the typed seed and its limit, the backend option, Stop, Stopped and Continue, progress bars and the computing chip, the note on a refused point, trials-only field and provisional samples, display-only while following, tab gating, default off.

  • examples/vaccination-campaign.test.ts, webgpu/compilation-report.test.ts:

    GPU eligibility and readiness of the example, its default marking, and a seeded check that the cost at the valley is below the domain's boundary points.

  • Storybook DOM probes and screenshots:

    SirCpu streaming, Stop at step 2 then Continue +4 to Complete 6 / 6, a hopeless pick stopping after 8 runs, the best point climbing to 100 runs.

How to test

  • yarn workspace @hashintel/petrinaut dev
  • Simulate > Browser optimizer (real) > SirCpu

    Expect the runtime to download once, then steps as dots on the Surface, the field filling in, the metric line streaming

  • Stop

    Expect Stopped, the navigation on the best step, refinement to 100 runs

  • Continue

    Expect steps resuming with continued numbering, then Complete

  • Click a corner of the Surface

    Expect "0 of 8 runs — refining" then "8 runs · cannot beat the best"

  • Click next to the best dot

    Expect refinement climbing to 100 runs and the metric line restreaming

  • Simulate > Browser optimizer (real) > VaccinationCampaign

    Expect a valley on the Surface with the best dot on its floor

  • Simulate > ViewOptimizationDrawer stories

    Expect the running, complete, stopped, fallback and remote variants to render

@vercel

vercel Bot commented Sep 3, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hash Ready Ready Preview Sep 8, 2026 2:44am UTC
petrinaut Ready Ready Preview Sep 8, 2026 2:44am UTC
petrinaut-docs Ready Ready Preview Sep 8, 2026 2:44am UTC
1 Skipped Deployment
Project Deployment Actions Updated
hashdotdesign-tokens Ignored Ignored Preview Sep 8, 2026 2:44am UTC

Request Review

@github-actions github-actions Bot added area/infra Relates to version control, CI, CD or IaC (area) area/libs Relates to first-party libraries/crates/packages (area) type/eng > frontend Owned by the @frontend team area/apps > hash.design Affects the `hash.design` design site (app) labels Sep 3, 2026
@kube kube self-assigned this Sep 3, 2026
@kube
kube force-pushed the claude/web-optimizer-react-channel branch from 06f4f63 to 9dfc3eb Compare September 3, 2026 00:33
@kube
kube force-pushed the claude/web-optimizer-react-channel branch 2 times, most recently from fe9bfc0 to 2f1fd14 Compare September 3, 2026 05:40
@kube
kube force-pushed the claude/web-optimizer-react-channel branch from 2f1fd14 to dc659ba Compare September 3, 2026 22:00
@kube
kube force-pushed the claude/web-optimizer-react-channel branch from dc659ba to fc5cf7b Compare September 3, 2026 22:23
@cursor

cursor Bot commented Sep 8, 2026

Copy link
Copy Markdown

PR Summary

High Risk
Large cross-cutting change to optimization lifecycle, experiment compute queuing, and backend fallback; incorrect trial/refinement or cancellation handling could skew results or leak workers, though coverage is extensive.

Overview
Adds in-browser optimization behind the experimental In-browser optimization setting: the host can supply a connected optimization source whose trials run locally instead of only a remote capability.

The experiments layer gains runDetachedObjective, which streams Monte Carlo batches for a frozen study snapshot on CPU or WebGPU (with backend selection, per-study compile cache, queued runs per queueKey, pinned seeds on CPU, and structured failure/cancel outcomes). Sweeps reuse a shared batch registry and throttle helpers moved under experiments/shared.

The optimizations stack wires a channel that maps each optimizer trial to one detached run (objective from per-run means or last frames; failures become pruned trials). OptimizationsProvider tracks connected study state—navigation, follow-trials, live selection, in-flight trials, activity, stop/continue via extendOptimization, and point refinement up to 100 runs—while remote studies stay unchanged. Public exports expand for connected optimization types.

Slider in @hashintel/ds-components supports disabled. @hashintel/petrinaut-core adds the Vaccination Campaign optimization example (SIR + policy levers, interior cost minimum, GPU-ready) with tests and docs.

Reviewed by Cursor Bugbot for commit b058535. Bugbot is set up for automated code reviews on this repo. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread libs/@hashintel/petrinaut/src/react/optimizations/provider.tsx

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Surface caching, best-point refinement, user documentation, and package changesets contain unresolved correctness issues.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds experimental in-browser optimization to Petrinaut, routing optimizer trials through the experiments backend with streaming, refinement, navigation, and continuation support.

Changes:

  • Adds connected optimizer lifecycle, feature gating, detached objective execution, and refinement.
  • Introduces connected-study UI, shared experiment components, and surface visualization enhancements.
  • Adds the Vaccination Campaign example, tests, stories, documentation, and package changeset.
File summaries
File Description
viewport-settings-dialog.tsx Adds the experimental setting.
viewport-settings-dialog.test.tsx Tests its default value.
simulate-view.tsx Gates connected optimization UI.
simulate-view.test.tsx Tests optimization tab gating.
use-gpu-availability.ts Extracts GPU eligibility analysis.
surface-frame.tsx Generalizes surface captions.
summary-strip.tsx Adds shared summary presentation.
metric-tiles.tsx Extracts reusable metric tiles.
format-value.ts Centralizes value formatting.
compute-backend-toggle.tsx Adds shared backend selection UI.
compute-backend-badge.tsx Adds shared backend status UI.
compute-activity.tsx Generalizes compute progress display.
study-summary-strip.tsx Summarizes connected study progress.
study-summary-strip.test.ts Tests step-progress descriptions.
steps-table.tsx Displays recent optimization steps.
study-progress.ts Derives study progress and activity.
optimization-navigator.tsx Adds connected parameter navigation.
optimization-metrics.tsx Displays streamed objective metrics.
navigator-band.tsx Composes connected navigation controls.
continue-control.tsx Adds study continuation controls.
view-optimization-drawer.stories.tsx Adds drawer variants.
optimizations-view.tsx Reuses shared formatting and statuses.
use-study-surface-walk.ts Samples remote study surfaces.
sample-study-cell.ts Caches and samples surface cells.
navigation-slice.ts Resolves surface slices and keys.
optimization-surface.stories.tsx Adds connected surface stories.
optimization-status.ts Distinguishes stopped connected studies.
optimization-seed.ts Generates and validates study seeds.
browser-optimizer.stories.tsx Adds real browser optimizer stories.
experiment-metrics.tsx Adopts shared metric tiles.
view-experiment-drawer.tsx Adopts the shared backend badge.
sweep-surface.tsx Uses generalized surface APIs.
experiments/shared/format-number.ts Removes superseded formatter.
experiments-story-fixtures.tsx Adds streaming objective fixtures.
bin-histogram-canvas.tsx Uses shared fixed formatting.
frame-popover.tsx Uses shared fixed formatting.
experiment-metric-timeline.tsx Supports fixed-size metric charts.
create-experiment-drawer.tsx Reuses shared GPU controls.
create-experiment-drawer.test.tsx Updates provider fixtures.
editor-view.tsx Lists the new example.
paint-field.ts Adds marker variants and sparse-field behavior.
contour-field.test.ts Tests irregular sparse interpolation.
contour-surface.tsx Supports display-only plots and marker options.
user-settings-provider.tsx Persists the new feature flag.
user-settings-context.ts Defines the feature setting.
use-optimization-source.ts Centralizes source gating.
surface-grid.ts Adds navigation key/value helpers.
sir-optimization-input.fixtures.ts Adds shared SIR test input.
objective-estimate.ts Estimates objective uncertainty.
objective-estimate.test.ts Tests refinement stopping logic.
point-refinement.ts Implements the refinement ladder.
fake-detached-objective-runs.fixtures.ts Adds controllable run fixtures.
optimizations/context.ts Expands connected study state and actions.
trial-outcome.ts Converts experiment outcomes to trial outcomes.
create-optimization-channel.ts Connects trials to experiment runs.
create-optimization-channel.test.ts Tests channel execution behavior.
optimization-context.ts Accepts connected optimization sources.
react/index.ts Exports connected optimization APIs.
sweep-session/README.md Updates shared-module architecture.
sweep-session/batch-registry.ts Removes the old private registry.
sweep-session/batch-registry.test.ts Removes superseded tests.
sweep-session.ts Adopts the shared registry and throttle.
shared/throttle.ts Adds reusable throttling.
shared/error-message.ts Adds shared error normalization.
shared/batch-registry.ts Adds the generic batch registry.
shared/batch-registry.test.ts Tests generic registry behavior.
experiments/provider.tsx Exposes streaming detached runs.
experiments/context.ts Defines detached run contracts.
main.ts Exports connected optimization types.
docs/examples.md Documents Vaccination Campaign.
compilation-report.test.ts Includes the new GPU-ready example.
vaccination-campaign.test.ts Tests example behavior and eligibility.
examples/index.ts Exports Vaccination Campaign.
examples.test.ts Adds the example to validation tests.
ai.ts Adds the example to documentation summaries.
Slider/slider.tsx Adds disabled slider support.
.changeset/connected-optimizer-source.md Records consumer-facing changes.
Review details
  • Files reviewed: 96/97 changed files
  • Comments generated: 4
  • Review effort level: Balanced

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .changeset/connected-optimizer-source.md Outdated
kube added 6 commits September 8, 2026 03:54
…orker kept

A worker crash or a study error settles the run, but the host's trial
evaluations kept a live signal and simulated on for outcomes no one would
read. Every terminal event now aborts the segment's controller. A crash also
took the studies kept for finished runs with it, so a later continuation
started a fresh worker only to fail with study_failed; those runs now lose
their resumable status and the continuation is refused with the reason.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread libs/@hashintel/petrinaut/src/react/optimizations/provider.tsx
kube added 11 commits September 8, 2026 04:14
A host had no way to tell a stopped study the worker kept from a first
segment cancelled before it reached the worker, so it offered Continue for
studies that were never created and the extension was refused. Terminal
events now carry `resumable`: true when the study stays in the worker, false
when the run failed or its segment never started. A remote service omits it.
The study seed masks a 32-bit crypto draw instead of reducing it modulo the
range. Stop on a connected study whose run has no id yet aborts the creation
so its run is cancelled where it was made. A point stopped early as unable to
beat the best resumes its ladder once it is the best. The surface cell cache
keys on the axes as well as the positions, so swapped axes never reuse each
other's samples. Changesets are one per package, with one for the Vaccination
Campaign example in petrinaut-core.
A cancelled terminal marked every connected study resumable while its local
study object existed, so a first segment stopped before it reached the worker,
as while the runtime loads or another study holds the worker, showed Continue
and then had its extension refused. The record's resumable flag now follows
the terminal event's, which the runtime sets from whether the study exists.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 5b6ce53. Configure here.

The refinement restarts when a point's isBest changes, but nothing asked it to
once the study had parked: a trial landing on the parked point and becoming
the best left the point stopped at its early-stop note. The connected study
now compares the best point's key before and after each trial event and, while
the navigation is parked, refines the point again; settling does the same for
a parked point instead of only settling a following study.
@hash-release hash-release Bot mentioned this pull request Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/apps > hash.design Affects the `hash.design` design site (app) area/deps Relates to third-party dependencies (area) area/infra Relates to version control, CI, CD or IaC (area) area/libs Relates to first-party libraries/crates/packages (area) type/eng > backend Owned by the @backend team type/eng > frontend Owned by the @frontend team

Development

Successfully merging this pull request may close these issues.

4 participants